{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Graph Learning and Graph Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will be creating a graph regularized version for a topic classification task. The task is to classify paper depending on their content. However in order to do so, we will also use the information encoded in the citation network that relates documents among each other. Of course, we do know that this kind of information is indeed powerful as papers belonging to the same subject tend to reference each other.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we will be using the Cora dataset available in the stellargraph library "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stellargraph import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = datasets.Cora()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config Completer.use_jedi = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_index = {\n",
    "      'Case_Based': 0,\n",
    "      'Genetic_Algorithms': 1,\n",
    "      'Neural_Networks': 2,\n",
    "      'Probabilistic_Methods': 3,\n",
    "      'Reinforcement_Learning': 4,\n",
    "      'Rule_Learning': 5,\n",
    "      'Theory': 6,\n",
    "  }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "G, labels = dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the Dataset object where we will both include information of the targeted sample (node) and its neighbors. In the following we will also allow to control the number of labelling instances to be used, in order to reproduce and evaluate the classification performance in a semi-supervised setting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import preprocessing, feature_extraction, model_selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.train import Example, Features, Feature, Int64List, BytesList, FloatList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "GRAPH_PREFIX=\"NL_nbr\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _int64_feature(*value):\n",
    "    \"\"\"Returns int64 tf.train.Feature from a bool / enum / int / uint.\"\"\"\n",
    "    return Feature(int64_list=Int64List(value=list(value)))\n",
    "\n",
    "def _bytes_feature(value):\n",
    "    \"\"\"Returns bytes tf.train.Feature from a string.\"\"\"\n",
    "    return Feature(\n",
    "        bytes_list=BytesList(value=[value.encode('utf-8')])\n",
    "    )\n",
    "\n",
    "def _float_feature(*value):\n",
    "    return Feature(float_list=FloatList(value=list(value)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import reduce\n",
    "from typing import List, Tuple\n",
    "import pandas as pd\n",
    "import six\n",
    "\n",
    "def addFeatures(x, y):\n",
    "    res = Features()\n",
    "    res.CopyFrom(x)\n",
    "    res.MergeFrom(y)\n",
    "    return res\n",
    "\n",
    "def neighborFeatures(features: Features, weight: float, prefix: str):\n",
    "    data = {f\"{prefix}_weight\": _float_feature(weight)}\n",
    "    for name, feature in six.iteritems(features.feature):\n",
    "        data[f\"{prefix}_{name}\"] = feature \n",
    "    return Features(feature=data)\n",
    "\n",
    "def neighborsFeatures(neighbors: List[Tuple[Features, float]]):\n",
    "    return reduce(\n",
    "        addFeatures, \n",
    "        [neighborFeatures(sample, weight, f\"{GRAPH_PREFIX}_{ith}\") for ith, (sample, weight) in enumerate(neighbors)],\n",
    "        Features()\n",
    "    )\n",
    "\n",
    "def getNeighbors(idx, adjMatrix, topn=5):\n",
    "    weights = adjMatrix.loc[idx]\n",
    "    return weights[weights>0].sort_values(ascending=False).head(topn).to_dict()\n",
    "    \n",
    "\n",
    "def semisupervisedDataset(G, labels, ratio=0.2, topn=5):\n",
    "    n = int(np.round(len(labels)*ratio))\n",
    "    \n",
    "    labelled, unlabelled = model_selection.train_test_split(\n",
    "        labels, train_size=n, test_size=None, stratify=labels\n",
    "    )\n",
    "    \n",
    "    adjMatrix = pd.DataFrame.sparse.from_spmatrix(G.to_adjacency_matrix(), index=G.nodes(), columns=G.nodes())\n",
    "    \n",
    "    features = pd.DataFrame(G.node_features(), index=G.nodes())\n",
    "    \n",
    "    dataset = {\n",
    "        index: Features(feature = {\n",
    "            #\"id\": _bytes_feature(str(index)), \n",
    "            \"id\": _int64_feature(index),\n",
    "            \"words\": _float_feature(*[float(x) for x in features.loc[index].values]), \n",
    "            \"label\": _int64_feature(label_index[label])\n",
    "        })\n",
    "        for index, label in pd.concat([labelled, unlabelled]).items()\n",
    "    }\n",
    "    \n",
    "    trainingSet = [\n",
    "        Example(features=addFeatures(\n",
    "            dataset[exampleId], \n",
    "            neighborsFeatures(\n",
    "                [(dataset[nodeId], weight) for nodeId, weight in getNeighbors(exampleId, adjMatrix, topn).items()]\n",
    "            )\n",
    "        ))\n",
    "        for exampleId in labelled.index\n",
    "    ]\n",
    "    \n",
    "    testSet = [Example(features=dataset[exampleId]) for exampleId in unlabelled.index]\n",
    "\n",
    "    serializer = lambda _list: [e.SerializeToString() for e in _list]\n",
    "    \n",
    "    return serializer(trainingSet), serializer(testSet)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the dataset into a training set and a test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainingSet, testSet = semisupervisedDataset(G, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.data import Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabularySize = 1433"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "neighbors=2\n",
    "defaultWord = tf.constant(0, dtype=tf.float32, shape=[vocabularySize])\n",
    "\n",
    "def parseExample(example, training=True):\n",
    "    schema = {\n",
    "        'words': tf.io.FixedLenFeature([vocabularySize], tf.float32, default_value=defaultWord),\n",
    "        'label': tf.io.FixedLenFeature((), tf.int64, default_value=-1)\n",
    "    }\n",
    "    \n",
    "    if training is True:\n",
    "        for i in range(neighbors):\n",
    "            name = f\"{GRAPH_PREFIX}_{i}\"\n",
    "            schema[f\"{name}_weight\"] = tf.io.FixedLenFeature([1], tf.float32, default_value=[0.0])\n",
    "            schema[f\"{name}_words\"] = tf.io.FixedLenFeature([vocabularySize], tf.float32, default_value=defaultWord)\n",
    "    \n",
    "    features = tf.io.parse_single_example(example, schema)\n",
    "    \n",
    "    label = features.pop(\"label\")\n",
    "    return features, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sampleGenerator(dataset):\n",
    "    def wrapper():\n",
    "        for example in dataset:\n",
    "            yield example\n",
    "    return wrapper\n",
    "            \n",
    "myTrain = Dataset \\\n",
    "    .from_generator(sampleGenerator(trainingSet), output_types=tf.string, output_shapes=()) \\\n",
    "    .map(lambda x: parseExample(x, True))\n",
    "\n",
    "myTest = Dataset \\\n",
    "    .from_generator(sampleGenerator(testSet), output_types=tf.string, output_shapes=()) \\\n",
    "    .map(lambda x: parseExample(x, False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'NL_nbr_0_weight': <tf.Tensor: shape=(10, 1), dtype=float32, numpy=\n",
      "array([[2.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [2.],\n",
      "       [1.],\n",
      "       [1.]], dtype=float32)>, 'NL_nbr_0_words': <tf.Tensor: shape=(10, 1433), dtype=float32, numpy=\n",
      "array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       ...,\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 1., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>, 'NL_nbr_1_weight': <tf.Tensor: shape=(10, 1), dtype=float32, numpy=\n",
      "array([[1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.]], dtype=float32)>, 'NL_nbr_1_words': <tf.Tensor: shape=(10, 1433), dtype=float32, numpy=\n",
      "array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       ...,\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>, 'words': <tf.Tensor: shape=(10, 1433), dtype=float32, numpy=\n",
      "array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       ...,\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>}\n",
      "tf.Tensor([1 1 4 1 0 5 2 1 6 3], shape=(10,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for features, labels in myTrain.batch(10).take(1):\n",
    "    print(features)\n",
    "    print(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'words': <tf.Tensor: shape=(10, 1433), dtype=float32, numpy=\n",
      "array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       ...,\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)>}\n",
      "tf.Tensor([1 1 3 3 2 4 5 3 2 6], shape=(10,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for features, labels in myTest.batch(10).take(1):\n",
    "    print(features)\n",
    "    print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the model that we will use to classify the documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [50, 50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"Creates a functional API-based multi-layer perceptron model.\"\"\"\n",
    "def create_model(num_units):\n",
    "    inputs = tf.keras.Input(\n",
    "          shape=(vocabularySize,), dtype='float32', name='words'\n",
    "    )\n",
    "\n",
    "    # outputs = tf.keras.layers.Dense(len(label_index), activation='softmax')(inputs)\n",
    "\n",
    "    cur_layer =  inputs\n",
    "\n",
    "    for num_units in layers:\n",
    "        cur_layer = tf.keras.layers.Dense(num_units, activation='relu')(cur_layer)\n",
    "        cur_layer = tf.keras.layers.Dropout(0.8)(cur_layer)\n",
    "\n",
    "    outputs = tf.keras.layers.Dense(len(label_index), activation='softmax')(cur_layer)\n",
    "\n",
    "    return tf.keras.Model(inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vanilla Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first train a simple, vanilla version that does not use the citation network information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model([50, 50])\n",
    "\n",
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "words (InputLayer)           [(None, 1433)]            0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 50)                71700     \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 7)                 357       \n",
      "=================================================================\n",
      "Total params: 74,607\n",
      "Trainable params: 74,607\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/deusebio/.pyenv/versions/3.7.6/envs/ml-book-4/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py:595: UserWarning: Input dict contained keys ['NL_nbr_0_weight', 'NL_nbr_0_words', 'NL_nbr_1_weight', 'NL_nbr_1_words'] which did not match any model input. They will be ignored by the model.\n",
      "  [n for n in tensors.keys() if n not in ref_input_names])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/5 [==============================] - 2s 323ms/step - loss: 1.9820 - accuracy: 0.1390 - val_loss: 1.9311 - val_accuracy: 0.2022\n",
      "Epoch 2/200\n",
      "5/5 [==============================] - 0s 114ms/step - loss: 1.9699 - accuracy: 0.1779 - val_loss: 1.9219 - val_accuracy: 0.2373\n",
      "Epoch 3/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.9632 - accuracy: 0.1964 - val_loss: 1.9135 - val_accuracy: 0.2742\n",
      "Epoch 4/200\n",
      "5/5 [==============================] - 0s 106ms/step - loss: 2.0003 - accuracy: 0.1981 - val_loss: 1.9071 - val_accuracy: 0.2886\n",
      "Epoch 5/200\n",
      "5/5 [==============================] - 1s 117ms/step - loss: 1.9681 - accuracy: 0.1810 - val_loss: 1.9010 - val_accuracy: 0.2941\n",
      "Epoch 6/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.8860 - accuracy: 0.2400 - val_loss: 1.8951 - val_accuracy: 0.3010\n",
      "Epoch 7/200\n",
      "5/5 [==============================] - 1s 124ms/step - loss: 1.8809 - accuracy: 0.2603 - val_loss: 1.8888 - val_accuracy: 0.3052\n",
      "Epoch 8/200\n",
      "5/5 [==============================] - 0s 111ms/step - loss: 1.8710 - accuracy: 0.2564 - val_loss: 1.8819 - val_accuracy: 0.3038\n",
      "Epoch 9/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.8745 - accuracy: 0.2519 - val_loss: 1.8754 - val_accuracy: 0.3015\n",
      "Epoch 10/200\n",
      "5/5 [==============================] - 0s 111ms/step - loss: 1.8847 - accuracy: 0.2621 - val_loss: 1.8691 - val_accuracy: 0.3024\n",
      "Epoch 11/200\n",
      "5/5 [==============================] - 1s 117ms/step - loss: 1.8810 - accuracy: 0.2581 - val_loss: 1.8633 - val_accuracy: 0.3029\n",
      "Epoch 12/200\n",
      "5/5 [==============================] - 1s 124ms/step - loss: 1.8815 - accuracy: 0.2402 - val_loss: 1.8580 - val_accuracy: 0.3024\n",
      "Epoch 13/200\n",
      "5/5 [==============================] - 0s 109ms/step - loss: 1.8592 - accuracy: 0.2660 - val_loss: 1.8530 - val_accuracy: 0.3024\n",
      "Epoch 14/200\n",
      "5/5 [==============================] - 0s 106ms/step - loss: 1.8818 - accuracy: 0.2764 - val_loss: 1.8481 - val_accuracy: 0.3024\n",
      "Epoch 15/200\n",
      "5/5 [==============================] - 0s 112ms/step - loss: 1.8602 - accuracy: 0.2645 - val_loss: 1.8436 - val_accuracy: 0.3029\n",
      "Epoch 16/200\n",
      "5/5 [==============================] - 0s 110ms/step - loss: 1.8200 - accuracy: 0.2913 - val_loss: 1.8386 - val_accuracy: 0.3024\n",
      "Epoch 17/200\n",
      "5/5 [==============================] - 0s 105ms/step - loss: 1.7878 - accuracy: 0.2694 - val_loss: 1.8328 - val_accuracy: 0.3024\n",
      "Epoch 18/200\n",
      "5/5 [==============================] - 0s 104ms/step - loss: 1.8208 - accuracy: 0.2823 - val_loss: 1.8262 - val_accuracy: 0.3019\n",
      "Epoch 19/200\n",
      "5/5 [==============================] - 0s 116ms/step - loss: 1.8273 - accuracy: 0.2808 - val_loss: 1.8200 - val_accuracy: 0.3019\n",
      "Epoch 20/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 1.8076 - accuracy: 0.2861 - val_loss: 1.8137 - val_accuracy: 0.3019\n",
      "Epoch 21/200\n",
      "5/5 [==============================] - 0s 112ms/step - loss: 1.7817 - accuracy: 0.2773 - val_loss: 1.8071 - val_accuracy: 0.3019\n",
      "Epoch 22/200\n",
      "5/5 [==============================] - 1s 148ms/step - loss: 1.7817 - accuracy: 0.2879 - val_loss: 1.7996 - val_accuracy: 0.3019\n",
      "Epoch 23/200\n",
      "5/5 [==============================] - 0s 106ms/step - loss: 1.8022 - accuracy: 0.2694 - val_loss: 1.7920 - val_accuracy: 0.3019\n",
      "Epoch 24/200\n",
      "5/5 [==============================] - 0s 113ms/step - loss: 1.7664 - accuracy: 0.2857 - val_loss: 1.7837 - val_accuracy: 0.3019\n",
      "Epoch 25/200\n",
      "5/5 [==============================] - 1s 118ms/step - loss: 1.7413 - accuracy: 0.3139 - val_loss: 1.7750 - val_accuracy: 0.3019\n",
      "Epoch 26/200\n",
      "5/5 [==============================] - 0s 111ms/step - loss: 1.7423 - accuracy: 0.2957 - val_loss: 1.7673 - val_accuracy: 0.3019\n",
      "Epoch 27/200\n",
      "5/5 [==============================] - 0s 104ms/step - loss: 1.7182 - accuracy: 0.3222 - val_loss: 1.7600 - val_accuracy: 0.3019\n",
      "Epoch 28/200\n",
      "5/5 [==============================] - 0s 109ms/step - loss: 1.7282 - accuracy: 0.3140 - val_loss: 1.7521 - val_accuracy: 0.3019\n",
      "Epoch 29/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 1.7342 - accuracy: 0.2928 - val_loss: 1.7451 - val_accuracy: 0.3019\n",
      "Epoch 30/200\n",
      "5/5 [==============================] - 0s 116ms/step - loss: 1.6762 - accuracy: 0.3217 - val_loss: 1.7368 - val_accuracy: 0.3019\n",
      "Epoch 31/200\n",
      "5/5 [==============================] - 0s 106ms/step - loss: 1.7440 - accuracy: 0.2937 - val_loss: 1.7286 - val_accuracy: 0.3019\n",
      "Epoch 32/200\n",
      "5/5 [==============================] - 1s 119ms/step - loss: 1.7002 - accuracy: 0.2902 - val_loss: 1.7214 - val_accuracy: 0.3024\n",
      "Epoch 33/200\n",
      "5/5 [==============================] - 0s 106ms/step - loss: 1.6856 - accuracy: 0.3105 - val_loss: 1.7143 - val_accuracy: 0.3029\n",
      "Epoch 34/200\n",
      "5/5 [==============================] - 1s 120ms/step - loss: 1.6947 - accuracy: 0.2990 - val_loss: 1.7079 - val_accuracy: 0.3033\n",
      "Epoch 35/200\n",
      "5/5 [==============================] - 0s 107ms/step - loss: 1.6785 - accuracy: 0.3159 - val_loss: 1.7024 - val_accuracy: 0.3038\n",
      "Epoch 36/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.6167 - accuracy: 0.3351 - val_loss: 1.6960 - val_accuracy: 0.3052\n",
      "Epoch 37/200\n",
      "5/5 [==============================] - 0s 114ms/step - loss: 1.6379 - accuracy: 0.3163 - val_loss: 1.6889 - val_accuracy: 0.3056\n",
      "Epoch 38/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.6286 - accuracy: 0.3426 - val_loss: 1.6814 - val_accuracy: 0.3066\n",
      "Epoch 39/200\n",
      "5/5 [==============================] - 1s 118ms/step - loss: 1.6328 - accuracy: 0.3559 - val_loss: 1.6738 - val_accuracy: 0.3084\n",
      "Epoch 40/200\n",
      "5/5 [==============================] - 1s 118ms/step - loss: 1.6194 - accuracy: 0.3266 - val_loss: 1.6667 - val_accuracy: 0.3126\n",
      "Epoch 41/200\n",
      "5/5 [==============================] - 1s 117ms/step - loss: 1.5999 - accuracy: 0.3031 - val_loss: 1.6612 - val_accuracy: 0.3181\n",
      "Epoch 42/200\n",
      "5/5 [==============================] - 0s 111ms/step - loss: 1.6033 - accuracy: 0.3178 - val_loss: 1.6555 - val_accuracy: 0.3246\n",
      "Epoch 43/200\n",
      "5/5 [==============================] - 0s 111ms/step - loss: 1.6016 - accuracy: 0.3283 - val_loss: 1.6490 - val_accuracy: 0.3338\n",
      "Epoch 44/200\n",
      "5/5 [==============================] - 0s 107ms/step - loss: 1.5466 - accuracy: 0.3435 - val_loss: 1.6403 - val_accuracy: 0.3430\n",
      "Epoch 45/200\n",
      "5/5 [==============================] - 0s 107ms/step - loss: 1.5700 - accuracy: 0.3411 - val_loss: 1.6300 - val_accuracy: 0.3500\n",
      "Epoch 46/200\n",
      "5/5 [==============================] - 1s 152ms/step - loss: 1.5677 - accuracy: 0.3146 - val_loss: 1.6208 - val_accuracy: 0.3587\n",
      "Epoch 47/200\n",
      "5/5 [==============================] - 1s 163ms/step - loss: 1.5848 - accuracy: 0.3527 - val_loss: 1.6121 - val_accuracy: 0.3652\n",
      "Epoch 48/200\n",
      "5/5 [==============================] - 1s 159ms/step - loss: 1.5458 - accuracy: 0.3726 - val_loss: 1.6040 - val_accuracy: 0.3703\n",
      "Epoch 49/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 1.5457 - accuracy: 0.3176 - val_loss: 1.5965 - val_accuracy: 0.3758\n",
      "Epoch 50/200\n",
      "5/5 [==============================] - 1s 143ms/step - loss: 1.5226 - accuracy: 0.3776 - val_loss: 1.5889 - val_accuracy: 0.3804\n",
      "Epoch 51/200\n",
      "5/5 [==============================] - 0s 113ms/step - loss: 1.5445 - accuracy: 0.3343 - val_loss: 1.5813 - val_accuracy: 0.3869\n",
      "Epoch 52/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 1.5416 - accuracy: 0.3672 - val_loss: 1.5740 - val_accuracy: 0.3938\n",
      "Epoch 53/200\n",
      "5/5 [==============================] - 0s 113ms/step - loss: 1.4925 - accuracy: 0.3735 - val_loss: 1.5666 - val_accuracy: 0.3984\n",
      "Epoch 54/200\n",
      "5/5 [==============================] - 1s 120ms/step - loss: 1.4956 - accuracy: 0.3615 - val_loss: 1.5591 - val_accuracy: 0.4035\n",
      "Epoch 55/200\n",
      "5/5 [==============================] - 0s 108ms/step - loss: 1.5306 - accuracy: 0.3399 - val_loss: 1.5520 - val_accuracy: 0.4090\n",
      "Epoch 56/200\n",
      "5/5 [==============================] - 0s 115ms/step - loss: 1.4512 - accuracy: 0.3820 - val_loss: 1.5445 - val_accuracy: 0.4155\n",
      "Epoch 57/200\n",
      "5/5 [==============================] - 1s 131ms/step - loss: 1.4307 - accuracy: 0.3891 - val_loss: 1.5374 - val_accuracy: 0.4183\n",
      "Epoch 58/200\n",
      "5/5 [==============================] - 1s 120ms/step - loss: 1.4430 - accuracy: 0.3657 - val_loss: 1.5300 - val_accuracy: 0.4247\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 59/200\n",
      "5/5 [==============================] - 1s 143ms/step - loss: 1.4438 - accuracy: 0.3667 - val_loss: 1.5221 - val_accuracy: 0.4280\n",
      "Epoch 60/200\n",
      "5/5 [==============================] - 0s 112ms/step - loss: 1.4592 - accuracy: 0.3633 - val_loss: 1.5149 - val_accuracy: 0.4326\n",
      "Epoch 61/200\n",
      "5/5 [==============================] - 1s 182ms/step - loss: 1.4056 - accuracy: 0.3988 - val_loss: 1.5073 - val_accuracy: 0.4367\n",
      "Epoch 62/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.4253 - accuracy: 0.3948 - val_loss: 1.5000 - val_accuracy: 0.4414\n",
      "Epoch 63/200\n",
      "5/5 [==============================] - 1s 149ms/step - loss: 1.3744 - accuracy: 0.4412 - val_loss: 1.4925 - val_accuracy: 0.4460\n",
      "Epoch 64/200\n",
      "5/5 [==============================] - 1s 133ms/step - loss: 1.3908 - accuracy: 0.4052 - val_loss: 1.4845 - val_accuracy: 0.4478\n",
      "Epoch 65/200\n",
      "5/5 [==============================] - 1s 119ms/step - loss: 1.3819 - accuracy: 0.4087 - val_loss: 1.4763 - val_accuracy: 0.4538\n",
      "Epoch 66/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.3798 - accuracy: 0.4137 - val_loss: 1.4688 - val_accuracy: 0.4566\n",
      "Epoch 67/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.3816 - accuracy: 0.4319 - val_loss: 1.4618 - val_accuracy: 0.4580\n",
      "Epoch 68/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 1.3548 - accuracy: 0.4387 - val_loss: 1.4549 - val_accuracy: 0.4594\n",
      "Epoch 69/200\n",
      "5/5 [==============================] - 1s 144ms/step - loss: 1.3364 - accuracy: 0.4546 - val_loss: 1.4484 - val_accuracy: 0.4608\n",
      "Epoch 70/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.3729 - accuracy: 0.4436 - val_loss: 1.4427 - val_accuracy: 0.4612\n",
      "Epoch 71/200\n",
      "5/5 [==============================] - 1s 142ms/step - loss: 1.3252 - accuracy: 0.4525 - val_loss: 1.4366 - val_accuracy: 0.4658\n",
      "Epoch 72/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.3500 - accuracy: 0.4421 - val_loss: 1.4302 - val_accuracy: 0.4695\n",
      "Epoch 73/200\n",
      "5/5 [==============================] - 1s 121ms/step - loss: 1.2927 - accuracy: 0.4529 - val_loss: 1.4238 - val_accuracy: 0.4718\n",
      "Epoch 74/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 1.2843 - accuracy: 0.4799 - val_loss: 1.4185 - val_accuracy: 0.4760\n",
      "Epoch 75/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.2724 - accuracy: 0.5005 - val_loss: 1.4128 - val_accuracy: 0.4829\n",
      "Epoch 76/200\n",
      "5/5 [==============================] - 1s 119ms/step - loss: 1.3129 - accuracy: 0.4651 - val_loss: 1.4083 - val_accuracy: 0.4848\n",
      "Epoch 77/200\n",
      "5/5 [==============================] - 1s 121ms/step - loss: 1.2618 - accuracy: 0.4697 - val_loss: 1.4047 - val_accuracy: 0.4861\n",
      "Epoch 78/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.2413 - accuracy: 0.5125 - val_loss: 1.4003 - val_accuracy: 0.4885\n",
      "Epoch 79/200\n",
      "5/5 [==============================] - 1s 118ms/step - loss: 1.2087 - accuracy: 0.5341 - val_loss: 1.3955 - val_accuracy: 0.4903\n",
      "Epoch 80/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.2851 - accuracy: 0.4639 - val_loss: 1.3900 - val_accuracy: 0.4935\n",
      "Epoch 81/200\n",
      "5/5 [==============================] - 1s 155ms/step - loss: 1.2209 - accuracy: 0.5177 - val_loss: 1.3837 - val_accuracy: 0.4986\n",
      "Epoch 82/200\n",
      "5/5 [==============================] - 1s 165ms/step - loss: 1.1996 - accuracy: 0.5319 - val_loss: 1.3780 - val_accuracy: 0.5005\n",
      "Epoch 83/200\n",
      "5/5 [==============================] - 1s 136ms/step - loss: 1.2368 - accuracy: 0.4949 - val_loss: 1.3740 - val_accuracy: 0.5046\n",
      "Epoch 84/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 1.2231 - accuracy: 0.5320 - val_loss: 1.3710 - val_accuracy: 0.5069\n",
      "Epoch 85/200\n",
      "5/5 [==============================] - 1s 124ms/step - loss: 1.2342 - accuracy: 0.5022 - val_loss: 1.3667 - val_accuracy: 0.5106\n",
      "Epoch 86/200\n",
      "5/5 [==============================] - 1s 121ms/step - loss: 1.2375 - accuracy: 0.4890 - val_loss: 1.3614 - val_accuracy: 0.5125\n",
      "Epoch 87/200\n",
      "5/5 [==============================] - 1s 126ms/step - loss: 1.2203 - accuracy: 0.5061 - val_loss: 1.3577 - val_accuracy: 0.5162\n",
      "Epoch 88/200\n",
      "5/5 [==============================] - 1s 120ms/step - loss: 1.2521 - accuracy: 0.5327 - val_loss: 1.3570 - val_accuracy: 0.5185\n",
      "Epoch 89/200\n",
      "5/5 [==============================] - 0s 117ms/step - loss: 1.1961 - accuracy: 0.5341 - val_loss: 1.3572 - val_accuracy: 0.5194\n",
      "Epoch 90/200\n",
      "5/5 [==============================] - 1s 123ms/step - loss: 1.1892 - accuracy: 0.5204 - val_loss: 1.3551 - val_accuracy: 0.5226\n",
      "Epoch 91/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.1887 - accuracy: 0.5302 - val_loss: 1.3544 - val_accuracy: 0.5268\n",
      "Epoch 92/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.1729 - accuracy: 0.5289 - val_loss: 1.3553 - val_accuracy: 0.5254\n",
      "Epoch 93/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 1.1671 - accuracy: 0.5114 - val_loss: 1.3555 - val_accuracy: 0.5277\n",
      "Epoch 94/200\n",
      "5/5 [==============================] - 1s 121ms/step - loss: 1.1611 - accuracy: 0.5413 - val_loss: 1.3543 - val_accuracy: 0.5277\n",
      "Epoch 95/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.1913 - accuracy: 0.5282 - val_loss: 1.3525 - val_accuracy: 0.5259\n",
      "Epoch 96/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.1435 - accuracy: 0.5480 - val_loss: 1.3524 - val_accuracy: 0.5263\n",
      "Epoch 97/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.1390 - accuracy: 0.5413 - val_loss: 1.3516 - val_accuracy: 0.5295\n",
      "Epoch 98/200\n",
      "5/5 [==============================] - 1s 120ms/step - loss: 1.1598 - accuracy: 0.5492 - val_loss: 1.3472 - val_accuracy: 0.5328\n",
      "Epoch 99/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 1.1754 - accuracy: 0.4923 - val_loss: 1.3409 - val_accuracy: 0.5360\n",
      "Epoch 100/200\n",
      "5/5 [==============================] - 1s 126ms/step - loss: 1.1441 - accuracy: 0.5521 - val_loss: 1.3364 - val_accuracy: 0.5397\n",
      "Epoch 101/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.1213 - accuracy: 0.5697 - val_loss: 1.3323 - val_accuracy: 0.5416\n",
      "Epoch 102/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.1414 - accuracy: 0.5274 - val_loss: 1.3311 - val_accuracy: 0.5420\n",
      "Epoch 103/200\n",
      "5/5 [==============================] - 1s 151ms/step - loss: 1.0863 - accuracy: 0.5567 - val_loss: 1.3315 - val_accuracy: 0.5439\n",
      "Epoch 104/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 1.0917 - accuracy: 0.5921 - val_loss: 1.3338 - val_accuracy: 0.5439\n",
      "Epoch 105/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.1084 - accuracy: 0.5622 - val_loss: 1.3379 - val_accuracy: 0.5416\n",
      "Epoch 106/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.0470 - accuracy: 0.5800 - val_loss: 1.3419 - val_accuracy: 0.5425\n",
      "Epoch 107/200\n",
      "5/5 [==============================] - 1s 124ms/step - loss: 1.0953 - accuracy: 0.5640 - val_loss: 1.3420 - val_accuracy: 0.5429\n",
      "Epoch 108/200\n",
      "5/5 [==============================] - 1s 133ms/step - loss: 1.0905 - accuracy: 0.5702 - val_loss: 1.3417 - val_accuracy: 0.5434\n",
      "Epoch 109/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.1185 - accuracy: 0.5661 - val_loss: 1.3425 - val_accuracy: 0.5457\n",
      "Epoch 110/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 1.0886 - accuracy: 0.5817 - val_loss: 1.3417 - val_accuracy: 0.5466\n",
      "Epoch 111/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 1.0223 - accuracy: 0.6096 - val_loss: 1.3427 - val_accuracy: 0.5476\n",
      "Epoch 112/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 1.0619 - accuracy: 0.5801 - val_loss: 1.3440 - val_accuracy: 0.5485\n",
      "Epoch 113/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 1.0970 - accuracy: 0.5693 - val_loss: 1.3435 - val_accuracy: 0.5489\n",
      "Epoch 114/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 1.0307 - accuracy: 0.5842 - val_loss: 1.3434 - val_accuracy: 0.5503\n",
      "Epoch 115/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 1.0729 - accuracy: 0.5749 - val_loss: 1.3409 - val_accuracy: 0.5512\n",
      "Epoch 116/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 1.0652 - accuracy: 0.5892 - val_loss: 1.3405 - val_accuracy: 0.5526\n",
      "Epoch 117/200\n",
      "5/5 [==============================] - 1s 146ms/step - loss: 1.0331 - accuracy: 0.5950 - val_loss: 1.3448 - val_accuracy: 0.5531\n",
      "Epoch 118/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.0661 - accuracy: 0.6041 - val_loss: 1.3518 - val_accuracy: 0.5536\n",
      "Epoch 119/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 1.0243 - accuracy: 0.6106 - val_loss: 1.3588 - val_accuracy: 0.5512\n",
      "Epoch 120/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 0.9959 - accuracy: 0.6434 - val_loss: 1.3643 - val_accuracy: 0.5508\n",
      "Epoch 121/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 1.0377 - accuracy: 0.6012 - val_loss: 1.3699 - val_accuracy: 0.5503\n",
      "Epoch 122/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 1.0587 - accuracy: 0.5726 - val_loss: 1.3722 - val_accuracy: 0.5512\n",
      "Epoch 123/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.0212 - accuracy: 0.5867 - val_loss: 1.3694 - val_accuracy: 0.5536\n",
      "Epoch 124/200\n",
      "5/5 [==============================] - 1s 189ms/step - loss: 1.0011 - accuracy: 0.6239 - val_loss: 1.3711 - val_accuracy: 0.5522\n",
      "Epoch 125/200\n",
      "5/5 [==============================] - 1s 141ms/step - loss: 0.9889 - accuracy: 0.6348 - val_loss: 1.3728 - val_accuracy: 0.5536\n",
      "Epoch 126/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 1.0428 - accuracy: 0.5786 - val_loss: 1.3751 - val_accuracy: 0.5522\n",
      "Epoch 127/200\n",
      "5/5 [==============================] - 1s 135ms/step - loss: 0.9908 - accuracy: 0.6201 - val_loss: 1.3732 - val_accuracy: 0.5545\n",
      "Epoch 128/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 1.0277 - accuracy: 0.5775 - val_loss: 1.3710 - val_accuracy: 0.5572\n",
      "Epoch 129/200\n",
      "5/5 [==============================] - 1s 165ms/step - loss: 0.9862 - accuracy: 0.6151 - val_loss: 1.3746 - val_accuracy: 0.5568\n",
      "Epoch 130/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 0.9637 - accuracy: 0.6345 - val_loss: 1.3762 - val_accuracy: 0.5572\n",
      "Epoch 131/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 0.9107 - accuracy: 0.6454 - val_loss: 1.3763 - val_accuracy: 0.5591\n",
      "Epoch 132/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.0057 - accuracy: 0.6081 - val_loss: 1.3782 - val_accuracy: 0.5605\n",
      "Epoch 133/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 0.9786 - accuracy: 0.6209 - val_loss: 1.3788 - val_accuracy: 0.5623\n",
      "Epoch 134/200\n",
      "5/5 [==============================] - 1s 123ms/step - loss: 1.0011 - accuracy: 0.5990 - val_loss: 1.3778 - val_accuracy: 0.5651\n",
      "Epoch 135/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 0.9755 - accuracy: 0.6250 - val_loss: 1.3808 - val_accuracy: 0.5660\n",
      "Epoch 136/200\n",
      "5/5 [==============================] - 1s 126ms/step - loss: 0.9770 - accuracy: 0.6175 - val_loss: 1.3842 - val_accuracy: 0.5660\n",
      "Epoch 137/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 0.9876 - accuracy: 0.6137 - val_loss: 1.3843 - val_accuracy: 0.5679\n",
      "Epoch 138/200\n",
      "5/5 [==============================] - 1s 136ms/step - loss: 0.9466 - accuracy: 0.6355 - val_loss: 1.3863 - val_accuracy: 0.5683\n",
      "Epoch 139/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.9569 - accuracy: 0.6377 - val_loss: 1.3873 - val_accuracy: 0.5679\n",
      "Epoch 140/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 0.9659 - accuracy: 0.6065 - val_loss: 1.3893 - val_accuracy: 0.5693\n",
      "Epoch 141/200\n",
      "5/5 [==============================] - 1s 139ms/step - loss: 0.9756 - accuracy: 0.6249 - val_loss: 1.3956 - val_accuracy: 0.5674\n",
      "Epoch 142/200\n",
      "5/5 [==============================] - 1s 131ms/step - loss: 0.9219 - accuracy: 0.6481 - val_loss: 1.4022 - val_accuracy: 0.5674\n",
      "Epoch 143/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.9725 - accuracy: 0.6032 - val_loss: 1.4084 - val_accuracy: 0.5683\n",
      "Epoch 144/200\n",
      "5/5 [==============================] - 1s 138ms/step - loss: 0.9968 - accuracy: 0.6080 - val_loss: 1.4106 - val_accuracy: 0.5693\n",
      "Epoch 145/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 0.9467 - accuracy: 0.6306 - val_loss: 1.4139 - val_accuracy: 0.5688\n",
      "Epoch 146/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.9845 - accuracy: 0.5994 - val_loss: 1.4185 - val_accuracy: 0.5683\n",
      "Epoch 147/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 0.9523 - accuracy: 0.6219 - val_loss: 1.4186 - val_accuracy: 0.5702\n",
      "Epoch 148/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 0.8707 - accuracy: 0.6598 - val_loss: 1.4205 - val_accuracy: 0.5716\n",
      "Epoch 149/200\n",
      "5/5 [==============================] - 1s 123ms/step - loss: 0.8485 - accuracy: 0.6838 - val_loss: 1.4247 - val_accuracy: 0.5716\n",
      "Epoch 150/200\n",
      "5/5 [==============================] - 1s 135ms/step - loss: 0.9404 - accuracy: 0.6376 - val_loss: 1.4308 - val_accuracy: 0.5702\n",
      "Epoch 151/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 0.9460 - accuracy: 0.6284 - val_loss: 1.4369 - val_accuracy: 0.5697\n",
      "Epoch 152/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 0.9460 - accuracy: 0.6174 - val_loss: 1.4409 - val_accuracy: 0.5725\n",
      "Epoch 153/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 0.9012 - accuracy: 0.6378 - val_loss: 1.4443 - val_accuracy: 0.5716\n",
      "Epoch 154/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 0.9226 - accuracy: 0.6226 - val_loss: 1.4550 - val_accuracy: 0.5720\n",
      "Epoch 155/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.9105 - accuracy: 0.6375 - val_loss: 1.4706 - val_accuracy: 0.5697\n",
      "Epoch 156/200\n",
      "5/5 [==============================] - 1s 157ms/step - loss: 0.8782 - accuracy: 0.6690 - val_loss: 1.4826 - val_accuracy: 0.5683\n",
      "Epoch 157/200\n",
      "5/5 [==============================] - 1s 129ms/step - loss: 0.9039 - accuracy: 0.6218 - val_loss: 1.4903 - val_accuracy: 0.5674\n",
      "Epoch 158/200\n",
      "5/5 [==============================] - 1s 135ms/step - loss: 0.9291 - accuracy: 0.6337 - val_loss: 1.4911 - val_accuracy: 0.5697\n",
      "Epoch 159/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 0.9176 - accuracy: 0.6366 - val_loss: 1.4898 - val_accuracy: 0.5693\n",
      "Epoch 160/200\n",
      "5/5 [==============================] - 1s 134ms/step - loss: 0.8564 - accuracy: 0.6516 - val_loss: 1.4861 - val_accuracy: 0.5706\n",
      "Epoch 161/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 0.8857 - accuracy: 0.6542 - val_loss: 1.4837 - val_accuracy: 0.5739\n",
      "Epoch 162/200\n",
      "5/5 [==============================] - 1s 128ms/step - loss: 0.8963 - accuracy: 0.6515 - val_loss: 1.4879 - val_accuracy: 0.5748\n",
      "Epoch 163/200\n",
      "5/5 [==============================] - 1s 125ms/step - loss: 0.8920 - accuracy: 0.6548 - val_loss: 1.4925 - val_accuracy: 0.5753\n",
      "Epoch 164/200\n",
      "5/5 [==============================] - 1s 123ms/step - loss: 0.8300 - accuracy: 0.6915 - val_loss: 1.5003 - val_accuracy: 0.5753\n",
      "Epoch 165/200\n",
      "5/5 [==============================] - 1s 133ms/step - loss: 0.9243 - accuracy: 0.6382 - val_loss: 1.5114 - val_accuracy: 0.5720\n",
      "Epoch 166/200\n",
      "5/5 [==============================] - 1s 122ms/step - loss: 0.8374 - accuracy: 0.6702 - val_loss: 1.5212 - val_accuracy: 0.5716\n",
      "Epoch 167/200\n",
      "5/5 [==============================] - 1s 127ms/step - loss: 0.8366 - accuracy: 0.6680 - val_loss: 1.5251 - val_accuracy: 0.5702\n",
      "Epoch 168/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.8265 - accuracy: 0.6612 - val_loss: 1.5247 - val_accuracy: 0.5734\n",
      "Epoch 169/200\n",
      "5/5 [==============================] - 1s 138ms/step - loss: 0.8960 - accuracy: 0.6386 - val_loss: 1.5208 - val_accuracy: 0.5757\n",
      "Epoch 170/200\n",
      "5/5 [==============================] - 1s 144ms/step - loss: 0.8747 - accuracy: 0.6373 - val_loss: 1.5191 - val_accuracy: 0.5771\n",
      "Epoch 171/200\n",
      "5/5 [==============================] - 1s 214ms/step - loss: 0.8566 - accuracy: 0.6471 - val_loss: 1.5159 - val_accuracy: 0.5803\n",
      "Epoch 172/200\n",
      "5/5 [==============================] - 1s 143ms/step - loss: 0.8520 - accuracy: 0.6547 - val_loss: 1.5122 - val_accuracy: 0.5799\n",
      "Epoch 173/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/5 [==============================] - 1s 241ms/step - loss: 0.8766 - accuracy: 0.6567 - val_loss: 1.5082 - val_accuracy: 0.5822\n",
      "Epoch 174/200\n",
      "5/5 [==============================] - 1s 219ms/step - loss: 0.7819 - accuracy: 0.7044 - val_loss: 1.5063 - val_accuracy: 0.5822\n",
      "Epoch 175/200\n",
      "5/5 [==============================] - 1s 184ms/step - loss: 0.8579 - accuracy: 0.6566 - val_loss: 1.5071 - val_accuracy: 0.5822\n",
      "Epoch 176/200\n",
      "5/5 [==============================] - 1s 186ms/step - loss: 0.8649 - accuracy: 0.6656 - val_loss: 1.5106 - val_accuracy: 0.5813\n",
      "Epoch 177/200\n",
      "5/5 [==============================] - 1s 193ms/step - loss: 0.8462 - accuracy: 0.6377 - val_loss: 1.5143 - val_accuracy: 0.5826\n",
      "Epoch 178/200\n",
      "5/5 [==============================] - 1s 175ms/step - loss: 0.8405 - accuracy: 0.6676 - val_loss: 1.5171 - val_accuracy: 0.5845\n",
      "Epoch 179/200\n",
      "5/5 [==============================] - 1s 183ms/step - loss: 0.8065 - accuracy: 0.6713 - val_loss: 1.5199 - val_accuracy: 0.5831\n",
      "Epoch 180/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 0.8850 - accuracy: 0.6369 - val_loss: 1.5232 - val_accuracy: 0.5831\n",
      "Epoch 181/200\n",
      "5/5 [==============================] - 1s 216ms/step - loss: 0.8796 - accuracy: 0.6546 - val_loss: 1.5259 - val_accuracy: 0.5826\n",
      "Epoch 182/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 0.7898 - accuracy: 0.6730 - val_loss: 1.5311 - val_accuracy: 0.5826\n",
      "Epoch 183/200\n",
      "5/5 [==============================] - 1s 159ms/step - loss: 0.7982 - accuracy: 0.6828 - val_loss: 1.5380 - val_accuracy: 0.5836\n",
      "Epoch 184/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 0.8291 - accuracy: 0.6792 - val_loss: 1.5468 - val_accuracy: 0.5836\n",
      "Epoch 185/200\n",
      "5/5 [==============================] - 1s 144ms/step - loss: 0.8375 - accuracy: 0.6788 - val_loss: 1.5529 - val_accuracy: 0.5840\n",
      "Epoch 186/200\n",
      "5/5 [==============================] - 1s 143ms/step - loss: 0.8144 - accuracy: 0.6788 - val_loss: 1.5538 - val_accuracy: 0.5840\n",
      "Epoch 187/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.8369 - accuracy: 0.6873 - val_loss: 1.5555 - val_accuracy: 0.5836\n",
      "Epoch 188/200\n",
      "5/5 [==============================] - 1s 130ms/step - loss: 0.8426 - accuracy: 0.6679 - val_loss: 1.5574 - val_accuracy: 0.5849\n",
      "Epoch 189/200\n",
      "5/5 [==============================] - 1s 144ms/step - loss: 0.7954 - accuracy: 0.6879 - val_loss: 1.5557 - val_accuracy: 0.5863\n",
      "Epoch 190/200\n",
      "5/5 [==============================] - 1s 131ms/step - loss: 0.7880 - accuracy: 0.6742 - val_loss: 1.5611 - val_accuracy: 0.5854\n",
      "Epoch 191/200\n",
      "5/5 [==============================] - 1s 172ms/step - loss: 0.8489 - accuracy: 0.6390 - val_loss: 1.5656 - val_accuracy: 0.5863\n",
      "Epoch 192/200\n",
      "5/5 [==============================] - 1s 175ms/step - loss: 0.8292 - accuracy: 0.6736 - val_loss: 1.5664 - val_accuracy: 0.5854\n",
      "Epoch 193/200\n",
      "5/5 [==============================] - 1s 151ms/step - loss: 0.7881 - accuracy: 0.6958 - val_loss: 1.5735 - val_accuracy: 0.5854\n",
      "Epoch 194/200\n",
      "5/5 [==============================] - 1s 145ms/step - loss: 0.8304 - accuracy: 0.6515 - val_loss: 1.5778 - val_accuracy: 0.5845\n",
      "Epoch 195/200\n",
      "5/5 [==============================] - 1s 217ms/step - loss: 0.7969 - accuracy: 0.6696 - val_loss: 1.5797 - val_accuracy: 0.5840\n",
      "Epoch 196/200\n",
      "5/5 [==============================] - 1s 183ms/step - loss: 0.7856 - accuracy: 0.6726 - val_loss: 1.5763 - val_accuracy: 0.5854\n",
      "Epoch 197/200\n",
      "5/5 [==============================] - 1s 170ms/step - loss: 0.7966 - accuracy: 0.6883 - val_loss: 1.5783 - val_accuracy: 0.5859\n",
      "Epoch 198/200\n",
      "5/5 [==============================] - 1s 147ms/step - loss: 0.8072 - accuracy: 0.6626 - val_loss: 1.5819 - val_accuracy: 0.5854\n",
      "Epoch 199/200\n",
      "5/5 [==============================] - 1s 135ms/step - loss: 0.7893 - accuracy: 0.6858 - val_loss: 1.5884 - val_accuracy: 0.5849\n",
      "Epoch 200/200\n",
      "5/5 [==============================] - 1s 154ms/step - loss: 0.7798 - accuracy: 0.6795 - val_loss: 1.5948 - val_accuracy: 0.5873\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14e4a9290>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(myTrain.batch(128), epochs=200, verbose=1, validation_data=myTest.batch(128),\n",
    "          callbacks=[TensorBoard(log_dir='/tmp/noRegularization')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Graph Regularized Version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the graph-regularized version that uses the citation network information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model = create_model([50, 50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import neural_structured_learning as nsl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_reg_config = nsl.configs.make_graph_reg_config(\n",
    "    max_neighbors=2,\n",
    "    multiplier=0.1,\n",
    "    distance_type=nsl.configs.DistanceType.L2,\n",
    "    sum_over_axis=-1)\n",
    "graph_reg_model = nsl.keras.GraphRegularization(base_model,\n",
    "                                                graph_reg_config)\n",
    "graph_reg_model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    metrics=['accuracy'])\n",
    "#graph_reg_model.fit(train_dataset, epochs=200, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/deusebio/.pyenv/versions/3.7.6/envs/ml-book-4/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/GraphRegularization/graph_loss/Reshape_1:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/GraphRegularization/graph_loss/Reshape:0\", shape=(None, 7), dtype=float32), dense_shape=Tensor(\"gradient_tape/GraphRegularization/graph_loss/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/5 [==============================] - 4s 328ms/step - loss: 2.1052 - accuracy: 0.1049 - scaled_graph_loss: 0.0056 - val_loss: 1.9455 - val_accuracy: 0.1440\n",
      "Epoch 2/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 2.0342 - accuracy: 0.1722 - scaled_graph_loss: 0.0047 - val_loss: 1.9367 - val_accuracy: 0.1934\n",
      "Epoch 3/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.9881 - accuracy: 0.1603 - scaled_graph_loss: 0.0039 - val_loss: 1.9293 - val_accuracy: 0.2355\n",
      "Epoch 4/200\n",
      "5/5 [==============================] - 1s 178ms/step - loss: 1.9918 - accuracy: 0.1816 - scaled_graph_loss: 0.0035 - val_loss: 1.9224 - val_accuracy: 0.2770\n",
      "Epoch 5/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 1.9551 - accuracy: 0.1983 - scaled_graph_loss: 0.0029 - val_loss: 1.9165 - val_accuracy: 0.2978\n",
      "Epoch 6/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.9745 - accuracy: 0.1918 - scaled_graph_loss: 0.0028 - val_loss: 1.9113 - val_accuracy: 0.3033\n",
      "Epoch 7/200\n",
      "5/5 [==============================] - 1s 170ms/step - loss: 1.9490 - accuracy: 0.1762 - scaled_graph_loss: 0.0023 - val_loss: 1.9068 - val_accuracy: 0.3075\n",
      "Epoch 8/200\n",
      "5/5 [==============================] - 1s 218ms/step - loss: 1.9262 - accuracy: 0.2016 - scaled_graph_loss: 0.0025 - val_loss: 1.9025 - val_accuracy: 0.3112\n",
      "Epoch 9/200\n",
      "5/5 [==============================] - 1s 208ms/step - loss: 1.9413 - accuracy: 0.2383 - scaled_graph_loss: 0.0023 - val_loss: 1.8983 - val_accuracy: 0.3066\n",
      "Epoch 10/200\n",
      "5/5 [==============================] - 1s 170ms/step - loss: 1.9136 - accuracy: 0.2278 - scaled_graph_loss: 0.0022 - val_loss: 1.8942 - val_accuracy: 0.3061\n",
      "Epoch 11/200\n",
      "5/5 [==============================] - 1s 154ms/step - loss: 1.9232 - accuracy: 0.2413 - scaled_graph_loss: 0.0022 - val_loss: 1.8900 - val_accuracy: 0.3066\n",
      "Epoch 12/200\n",
      "5/5 [==============================] - 1s 157ms/step - loss: 1.8885 - accuracy: 0.2682 - scaled_graph_loss: 0.0022 - val_loss: 1.8857 - val_accuracy: 0.3052\n",
      "Epoch 13/200\n",
      "5/5 [==============================] - 1s 192ms/step - loss: 1.8657 - accuracy: 0.2789 - scaled_graph_loss: 0.0021 - val_loss: 1.8813 - val_accuracy: 0.3052\n",
      "Epoch 14/200\n",
      "5/5 [==============================] - 1s 178ms/step - loss: 1.9115 - accuracy: 0.2158 - scaled_graph_loss: 0.0022 - val_loss: 1.8771 - val_accuracy: 0.3033\n",
      "Epoch 15/200\n",
      "5/5 [==============================] - 1s 174ms/step - loss: 1.8700 - accuracy: 0.2652 - scaled_graph_loss: 0.0020 - val_loss: 1.8725 - val_accuracy: 0.3029\n",
      "Epoch 16/200\n",
      "5/5 [==============================] - 1s 154ms/step - loss: 1.8883 - accuracy: 0.2607 - scaled_graph_loss: 0.0026 - val_loss: 1.8676 - val_accuracy: 0.3024\n",
      "Epoch 17/200\n",
      "5/5 [==============================] - 1s 141ms/step - loss: 1.8588 - accuracy: 0.2631 - scaled_graph_loss: 0.0025 - val_loss: 1.8625 - val_accuracy: 0.3019\n",
      "Epoch 18/200\n",
      "5/5 [==============================] - 1s 152ms/step - loss: 1.8774 - accuracy: 0.2593 - scaled_graph_loss: 0.0026 - val_loss: 1.8577 - val_accuracy: 0.3019\n",
      "Epoch 19/200\n",
      "5/5 [==============================] - 1s 187ms/step - loss: 1.8376 - accuracy: 0.2726 - scaled_graph_loss: 0.0028 - val_loss: 1.8530 - val_accuracy: 0.3019\n",
      "Epoch 20/200\n",
      "5/5 [==============================] - 1s 146ms/step - loss: 1.8590 - accuracy: 0.3045 - scaled_graph_loss: 0.0030 - val_loss: 1.8485 - val_accuracy: 0.3019\n",
      "Epoch 21/200\n",
      "5/5 [==============================] - 1s 204ms/step - loss: 1.8240 - accuracy: 0.2647 - scaled_graph_loss: 0.0028 - val_loss: 1.8439 - val_accuracy: 0.3019\n",
      "Epoch 22/200\n",
      "5/5 [==============================] - 1s 159ms/step - loss: 1.8541 - accuracy: 0.2940 - scaled_graph_loss: 0.0032 - val_loss: 1.8397 - val_accuracy: 0.3019\n",
      "Epoch 23/200\n",
      "5/5 [==============================] - 1s 208ms/step - loss: 1.8121 - accuracy: 0.3230 - scaled_graph_loss: 0.0034 - val_loss: 1.8350 - val_accuracy: 0.3019\n",
      "Epoch 24/200\n",
      "5/5 [==============================] - 1s 189ms/step - loss: 1.8325 - accuracy: 0.2986 - scaled_graph_loss: 0.0032 - val_loss: 1.8304 - val_accuracy: 0.3019\n",
      "Epoch 25/200\n",
      "5/5 [==============================] - 1s 170ms/step - loss: 1.8066 - accuracy: 0.2818 - scaled_graph_loss: 0.0032 - val_loss: 1.8254 - val_accuracy: 0.3019\n",
      "Epoch 26/200\n",
      "5/5 [==============================] - 1s 204ms/step - loss: 1.8389 - accuracy: 0.2880 - scaled_graph_loss: 0.0036 - val_loss: 1.8211 - val_accuracy: 0.3019\n",
      "Epoch 27/200\n",
      "5/5 [==============================] - 1s 208ms/step - loss: 1.8070 - accuracy: 0.3068 - scaled_graph_loss: 0.0034 - val_loss: 1.8168 - val_accuracy: 0.3024\n",
      "Epoch 28/200\n",
      "5/5 [==============================] - 1s 171ms/step - loss: 1.8070 - accuracy: 0.2909 - scaled_graph_loss: 0.0038 - val_loss: 1.8122 - val_accuracy: 0.3024\n",
      "Epoch 29/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 1.7793 - accuracy: 0.2947 - scaled_graph_loss: 0.0046 - val_loss: 1.8069 - val_accuracy: 0.3024\n",
      "Epoch 30/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.7738 - accuracy: 0.2886 - scaled_graph_loss: 0.0048 - val_loss: 1.8016 - val_accuracy: 0.3024\n",
      "Epoch 31/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.7596 - accuracy: 0.3011 - scaled_graph_loss: 0.0043 - val_loss: 1.7957 - val_accuracy: 0.3024\n",
      "Epoch 32/200\n",
      "5/5 [==============================] - 1s 261ms/step - loss: 1.7768 - accuracy: 0.3165 - scaled_graph_loss: 0.0053 - val_loss: 1.7900 - val_accuracy: 0.3024\n",
      "Epoch 33/200\n",
      "5/5 [==============================] - 1s 196ms/step - loss: 1.7419 - accuracy: 0.3019 - scaled_graph_loss: 0.0052 - val_loss: 1.7840 - val_accuracy: 0.3024\n",
      "Epoch 34/200\n",
      "5/5 [==============================] - 1s 177ms/step - loss: 1.7615 - accuracy: 0.2834 - scaled_graph_loss: 0.0060 - val_loss: 1.7783 - val_accuracy: 0.3024\n",
      "Epoch 35/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.7419 - accuracy: 0.3210 - scaled_graph_loss: 0.0057 - val_loss: 1.7730 - val_accuracy: 0.3024\n",
      "Epoch 36/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 1.7253 - accuracy: 0.3277 - scaled_graph_loss: 0.0061 - val_loss: 1.7670 - val_accuracy: 0.3024\n",
      "Epoch 37/200\n",
      "5/5 [==============================] - 1s 212ms/step - loss: 1.7170 - accuracy: 0.3362 - scaled_graph_loss: 0.0061 - val_loss: 1.7608 - val_accuracy: 0.3024\n",
      "Epoch 38/200\n",
      "5/5 [==============================] - 1s 212ms/step - loss: 1.7038 - accuracy: 0.3185 - scaled_graph_loss: 0.0064 - val_loss: 1.7548 - val_accuracy: 0.3024\n",
      "Epoch 39/200\n",
      "5/5 [==============================] - 1s 207ms/step - loss: 1.7044 - accuracy: 0.3301 - scaled_graph_loss: 0.0065 - val_loss: 1.7491 - val_accuracy: 0.3024\n",
      "Epoch 40/200\n",
      "5/5 [==============================] - 1s 152ms/step - loss: 1.7011 - accuracy: 0.3390 - scaled_graph_loss: 0.0069 - val_loss: 1.7428 - val_accuracy: 0.3029\n",
      "Epoch 41/200\n",
      "5/5 [==============================] - 1s 192ms/step - loss: 1.6931 - accuracy: 0.3415 - scaled_graph_loss: 0.0075 - val_loss: 1.7368 - val_accuracy: 0.3042\n",
      "Epoch 42/200\n",
      "5/5 [==============================] - 1s 159ms/step - loss: 1.7035 - accuracy: 0.3211 - scaled_graph_loss: 0.0074 - val_loss: 1.7310 - val_accuracy: 0.3038\n",
      "Epoch 43/200\n",
      "5/5 [==============================] - 1s 151ms/step - loss: 1.6884 - accuracy: 0.3293 - scaled_graph_loss: 0.0075 - val_loss: 1.7258 - val_accuracy: 0.3047\n",
      "Epoch 44/200\n",
      "5/5 [==============================] - 1s 148ms/step - loss: 1.7046 - accuracy: 0.3319 - scaled_graph_loss: 0.0079 - val_loss: 1.7206 - val_accuracy: 0.3047\n",
      "Epoch 45/200\n",
      "5/5 [==============================] - 1s 151ms/step - loss: 1.6325 - accuracy: 0.3369 - scaled_graph_loss: 0.0082 - val_loss: 1.7141 - val_accuracy: 0.3052\n",
      "Epoch 46/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 1.6682 - accuracy: 0.3413 - scaled_graph_loss: 0.0082 - val_loss: 1.7070 - val_accuracy: 0.3061\n",
      "Epoch 47/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 1.5976 - accuracy: 0.3434 - scaled_graph_loss: 0.0094 - val_loss: 1.6997 - val_accuracy: 0.3066\n",
      "Epoch 48/200\n",
      "5/5 [==============================] - 1s 183ms/step - loss: 1.6572 - accuracy: 0.3536 - scaled_graph_loss: 0.0093 - val_loss: 1.6933 - val_accuracy: 0.3084\n",
      "Epoch 49/200\n",
      "5/5 [==============================] - 1s 159ms/step - loss: 1.6508 - accuracy: 0.3540 - scaled_graph_loss: 0.0098 - val_loss: 1.6861 - val_accuracy: 0.3112\n",
      "Epoch 50/200\n",
      "5/5 [==============================] - 1s 148ms/step - loss: 1.6408 - accuracy: 0.3250 - scaled_graph_loss: 0.0097 - val_loss: 1.6793 - val_accuracy: 0.3149\n",
      "Epoch 51/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 1.6336 - accuracy: 0.3462 - scaled_graph_loss: 0.0092 - val_loss: 1.6736 - val_accuracy: 0.3190\n",
      "Epoch 52/200\n",
      "5/5 [==============================] - 1s 227ms/step - loss: 1.6309 - accuracy: 0.3321 - scaled_graph_loss: 0.0103 - val_loss: 1.6677 - val_accuracy: 0.3236\n",
      "Epoch 53/200\n",
      "5/5 [==============================] - 1s 271ms/step - loss: 1.6388 - accuracy: 0.3596 - scaled_graph_loss: 0.0111 - val_loss: 1.6628 - val_accuracy: 0.3287\n",
      "Epoch 54/200\n",
      "5/5 [==============================] - 1s 178ms/step - loss: 1.5818 - accuracy: 0.3541 - scaled_graph_loss: 0.0103 - val_loss: 1.6570 - val_accuracy: 0.3338\n",
      "Epoch 55/200\n",
      "5/5 [==============================] - 1s 272ms/step - loss: 1.5723 - accuracy: 0.3685 - scaled_graph_loss: 0.0109 - val_loss: 1.6503 - val_accuracy: 0.3398\n",
      "Epoch 56/200\n",
      "5/5 [==============================] - 1s 220ms/step - loss: 1.5742 - accuracy: 0.3803 - scaled_graph_loss: 0.0103 - val_loss: 1.6434 - val_accuracy: 0.3467\n",
      "Epoch 57/200\n",
      "5/5 [==============================] - 1s 200ms/step - loss: 1.5509 - accuracy: 0.3862 - scaled_graph_loss: 0.0114 - val_loss: 1.6370 - val_accuracy: 0.3578\n",
      "Epoch 58/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.5821 - accuracy: 0.3674 - scaled_graph_loss: 0.0117 - val_loss: 1.6305 - val_accuracy: 0.3638\n",
      "Epoch 59/200\n",
      "5/5 [==============================] - 1s 151ms/step - loss: 1.5852 - accuracy: 0.3616 - scaled_graph_loss: 0.0118 - val_loss: 1.6237 - val_accuracy: 0.3698\n",
      "Epoch 60/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.5499 - accuracy: 0.3611 - scaled_graph_loss: 0.0115 - val_loss: 1.6169 - val_accuracy: 0.3726\n",
      "Epoch 61/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.5457 - accuracy: 0.3744 - scaled_graph_loss: 0.0142 - val_loss: 1.6107 - val_accuracy: 0.3795\n",
      "Epoch 62/200\n",
      "5/5 [==============================] - 1s 132ms/step - loss: 1.5397 - accuracy: 0.3819 - scaled_graph_loss: 0.0127 - val_loss: 1.6045 - val_accuracy: 0.3892\n",
      "Epoch 63/200\n",
      "5/5 [==============================] - 1s 153ms/step - loss: 1.5438 - accuracy: 0.4087 - scaled_graph_loss: 0.0136 - val_loss: 1.5984 - val_accuracy: 0.3947\n",
      "Epoch 64/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 1.5161 - accuracy: 0.3815 - scaled_graph_loss: 0.0135 - val_loss: 1.5919 - val_accuracy: 0.4017\n",
      "Epoch 65/200\n",
      "5/5 [==============================] - 1s 143ms/step - loss: 1.5581 - accuracy: 0.3713 - scaled_graph_loss: 0.0134 - val_loss: 1.5854 - val_accuracy: 0.4086\n",
      "Epoch 66/200\n",
      "5/5 [==============================] - 1s 138ms/step - loss: 1.5347 - accuracy: 0.3858 - scaled_graph_loss: 0.0122 - val_loss: 1.5794 - val_accuracy: 0.4197\n",
      "Epoch 67/200\n",
      "5/5 [==============================] - 1s 204ms/step - loss: 1.4917 - accuracy: 0.4024 - scaled_graph_loss: 0.0141 - val_loss: 1.5733 - val_accuracy: 0.4326\n",
      "Epoch 68/200\n",
      "5/5 [==============================] - 1s 223ms/step - loss: 1.4804 - accuracy: 0.3927 - scaled_graph_loss: 0.0129 - val_loss: 1.5658 - val_accuracy: 0.4483\n",
      "Epoch 69/200\n",
      "5/5 [==============================] - 1s 252ms/step - loss: 1.4751 - accuracy: 0.4093 - scaled_graph_loss: 0.0138 - val_loss: 1.5586 - val_accuracy: 0.4575\n",
      "Epoch 70/200\n",
      "5/5 [==============================] - 1s 192ms/step - loss: 1.4870 - accuracy: 0.4230 - scaled_graph_loss: 0.0140 - val_loss: 1.5514 - val_accuracy: 0.4645\n",
      "Epoch 71/200\n",
      "5/5 [==============================] - 1s 213ms/step - loss: 1.4541 - accuracy: 0.4380 - scaled_graph_loss: 0.0146 - val_loss: 1.5447 - val_accuracy: 0.4686\n",
      "Epoch 72/200\n",
      "5/5 [==============================] - 1s 264ms/step - loss: 1.4528 - accuracy: 0.4193 - scaled_graph_loss: 0.0141 - val_loss: 1.5373 - val_accuracy: 0.4765\n",
      "Epoch 73/200\n",
      "5/5 [==============================] - 1s 199ms/step - loss: 1.4318 - accuracy: 0.4144 - scaled_graph_loss: 0.0177 - val_loss: 1.5302 - val_accuracy: 0.4820\n",
      "Epoch 74/200\n",
      "5/5 [==============================] - 1s 210ms/step - loss: 1.4495 - accuracy: 0.4282 - scaled_graph_loss: 0.0161 - val_loss: 1.5239 - val_accuracy: 0.4926\n",
      "Epoch 75/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 1.4085 - accuracy: 0.4447 - scaled_graph_loss: 0.0161 - val_loss: 1.5170 - val_accuracy: 0.4958\n",
      "Epoch 76/200\n",
      "5/5 [==============================] - 1s 171ms/step - loss: 1.3982 - accuracy: 0.4639 - scaled_graph_loss: 0.0155 - val_loss: 1.5103 - val_accuracy: 0.5000\n",
      "Epoch 77/200\n",
      "5/5 [==============================] - 1s 186ms/step - loss: 1.3995 - accuracy: 0.4773 - scaled_graph_loss: 0.0168 - val_loss: 1.5037 - val_accuracy: 0.5037\n",
      "Epoch 78/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 1.4244 - accuracy: 0.4426 - scaled_graph_loss: 0.0185 - val_loss: 1.4985 - val_accuracy: 0.5074\n",
      "Epoch 79/200\n",
      "5/5 [==============================] - 1s 194ms/step - loss: 1.4186 - accuracy: 0.4440 - scaled_graph_loss: 0.0163 - val_loss: 1.4933 - val_accuracy: 0.5102\n",
      "Epoch 80/200\n",
      "5/5 [==============================] - 1s 212ms/step - loss: 1.3805 - accuracy: 0.4560 - scaled_graph_loss: 0.0170 - val_loss: 1.4872 - val_accuracy: 0.5115\n",
      "Epoch 81/200\n",
      "5/5 [==============================] - 1s 204ms/step - loss: 1.3641 - accuracy: 0.4687 - scaled_graph_loss: 0.0173 - val_loss: 1.4800 - val_accuracy: 0.5166\n",
      "Epoch 82/200\n",
      "5/5 [==============================] - 1s 207ms/step - loss: 1.3796 - accuracy: 0.4717 - scaled_graph_loss: 0.0164 - val_loss: 1.4728 - val_accuracy: 0.5249\n",
      "Epoch 83/200\n",
      "5/5 [==============================] - 1s 203ms/step - loss: 1.4130 - accuracy: 0.4492 - scaled_graph_loss: 0.0177 - val_loss: 1.4667 - val_accuracy: 0.5314\n",
      "Epoch 84/200\n",
      "5/5 [==============================] - 1s 222ms/step - loss: 1.3239 - accuracy: 0.4916 - scaled_graph_loss: 0.0197 - val_loss: 1.4608 - val_accuracy: 0.5369\n",
      "Epoch 85/200\n",
      "5/5 [==============================] - 1s 220ms/step - loss: 1.3893 - accuracy: 0.4488 - scaled_graph_loss: 0.0186 - val_loss: 1.4551 - val_accuracy: 0.5434\n",
      "Epoch 86/200\n",
      "5/5 [==============================] - 1s 236ms/step - loss: 1.3161 - accuracy: 0.4909 - scaled_graph_loss: 0.0204 - val_loss: 1.4490 - val_accuracy: 0.5466\n",
      "Epoch 87/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 1.3434 - accuracy: 0.4966 - scaled_graph_loss: 0.0200 - val_loss: 1.4419 - val_accuracy: 0.5476\n",
      "Epoch 88/200\n",
      "5/5 [==============================] - 1s 250ms/step - loss: 1.3027 - accuracy: 0.5098 - scaled_graph_loss: 0.0196 - val_loss: 1.4357 - val_accuracy: 0.5489\n",
      "Epoch 89/200\n",
      "5/5 [==============================] - 1s 230ms/step - loss: 1.3019 - accuracy: 0.4970 - scaled_graph_loss: 0.0201 - val_loss: 1.4293 - val_accuracy: 0.5526\n",
      "Epoch 90/200\n",
      "5/5 [==============================] - 1s 248ms/step - loss: 1.2992 - accuracy: 0.4944 - scaled_graph_loss: 0.0205 - val_loss: 1.4230 - val_accuracy: 0.5559\n",
      "Epoch 91/200\n",
      "5/5 [==============================] - 1s 231ms/step - loss: 1.3239 - accuracy: 0.4901 - scaled_graph_loss: 0.0179 - val_loss: 1.4171 - val_accuracy: 0.5619\n",
      "Epoch 92/200\n",
      "5/5 [==============================] - 1s 221ms/step - loss: 1.3136 - accuracy: 0.5136 - scaled_graph_loss: 0.0196 - val_loss: 1.4112 - val_accuracy: 0.5669\n",
      "Epoch 93/200\n",
      "5/5 [==============================] - 1s 183ms/step - loss: 1.2639 - accuracy: 0.5288 - scaled_graph_loss: 0.0209 - val_loss: 1.4053 - val_accuracy: 0.5665\n",
      "Epoch 94/200\n",
      "5/5 [==============================] - 1s 205ms/step - loss: 1.2763 - accuracy: 0.5047 - scaled_graph_loss: 0.0209 - val_loss: 1.3995 - val_accuracy: 0.5665\n",
      "Epoch 95/200\n",
      "5/5 [==============================] - 1s 238ms/step - loss: 1.2617 - accuracy: 0.5052 - scaled_graph_loss: 0.0207 - val_loss: 1.3948 - val_accuracy: 0.5656\n",
      "Epoch 96/200\n",
      "5/5 [==============================] - 1s 218ms/step - loss: 1.2874 - accuracy: 0.5022 - scaled_graph_loss: 0.0226 - val_loss: 1.3906 - val_accuracy: 0.5697\n",
      "Epoch 97/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/5 [==============================] - 1s 256ms/step - loss: 1.2262 - accuracy: 0.5307 - scaled_graph_loss: 0.0216 - val_loss: 1.3858 - val_accuracy: 0.5702\n",
      "Epoch 98/200\n",
      "5/5 [==============================] - 1s 197ms/step - loss: 1.2362 - accuracy: 0.5532 - scaled_graph_loss: 0.0216 - val_loss: 1.3793 - val_accuracy: 0.5725\n",
      "Epoch 99/200\n",
      "5/5 [==============================] - 1s 262ms/step - loss: 1.2081 - accuracy: 0.5314 - scaled_graph_loss: 0.0236 - val_loss: 1.3731 - val_accuracy: 0.5748\n",
      "Epoch 100/200\n",
      "5/5 [==============================] - 1s 160ms/step - loss: 1.2115 - accuracy: 0.5213 - scaled_graph_loss: 0.0224 - val_loss: 1.3678 - val_accuracy: 0.5780\n",
      "Epoch 101/200\n",
      "5/5 [==============================] - 1s 153ms/step - loss: 1.1994 - accuracy: 0.5480 - scaled_graph_loss: 0.0230 - val_loss: 1.3620 - val_accuracy: 0.5785\n",
      "Epoch 102/200\n",
      "5/5 [==============================] - 1s 144ms/step - loss: 1.1956 - accuracy: 0.5351 - scaled_graph_loss: 0.0240 - val_loss: 1.3569 - val_accuracy: 0.5799\n",
      "Epoch 103/200\n",
      "5/5 [==============================] - 1s 154ms/step - loss: 1.2904 - accuracy: 0.4833 - scaled_graph_loss: 0.0225 - val_loss: 1.3529 - val_accuracy: 0.5803\n",
      "Epoch 104/200\n",
      "5/5 [==============================] - 1s 210ms/step - loss: 1.1866 - accuracy: 0.5519 - scaled_graph_loss: 0.0241 - val_loss: 1.3486 - val_accuracy: 0.5799\n",
      "Epoch 105/200\n",
      "5/5 [==============================] - 1s 184ms/step - loss: 1.2259 - accuracy: 0.5028 - scaled_graph_loss: 0.0225 - val_loss: 1.3449 - val_accuracy: 0.5822\n",
      "Epoch 106/200\n",
      "5/5 [==============================] - 1s 218ms/step - loss: 1.2135 - accuracy: 0.5383 - scaled_graph_loss: 0.0246 - val_loss: 1.3409 - val_accuracy: 0.5826\n",
      "Epoch 107/200\n",
      "5/5 [==============================] - 1s 215ms/step - loss: 1.1978 - accuracy: 0.5241 - scaled_graph_loss: 0.0244 - val_loss: 1.3366 - val_accuracy: 0.5831\n",
      "Epoch 108/200\n",
      "5/5 [==============================] - 1s 214ms/step - loss: 1.1885 - accuracy: 0.5566 - scaled_graph_loss: 0.0250 - val_loss: 1.3322 - val_accuracy: 0.5845\n",
      "Epoch 109/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.1876 - accuracy: 0.5501 - scaled_graph_loss: 0.0258 - val_loss: 1.3270 - val_accuracy: 0.5877\n",
      "Epoch 110/200\n",
      "5/5 [==============================] - 1s 172ms/step - loss: 1.1515 - accuracy: 0.5736 - scaled_graph_loss: 0.0251 - val_loss: 1.3238 - val_accuracy: 0.5896\n",
      "Epoch 111/200\n",
      "5/5 [==============================] - 1s 166ms/step - loss: 1.1388 - accuracy: 0.5336 - scaled_graph_loss: 0.0248 - val_loss: 1.3204 - val_accuracy: 0.5910\n",
      "Epoch 112/200\n",
      "5/5 [==============================] - 1s 176ms/step - loss: 1.1100 - accuracy: 0.5436 - scaled_graph_loss: 0.0266 - val_loss: 1.3165 - val_accuracy: 0.5928\n",
      "Epoch 113/200\n",
      "5/5 [==============================] - 1s 158ms/step - loss: 1.1504 - accuracy: 0.5861 - scaled_graph_loss: 0.0254 - val_loss: 1.3145 - val_accuracy: 0.5910\n",
      "Epoch 114/200\n",
      "5/5 [==============================] - 1s 214ms/step - loss: 1.1706 - accuracy: 0.5344 - scaled_graph_loss: 0.0262 - val_loss: 1.3123 - val_accuracy: 0.5905\n",
      "Epoch 115/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 1.1645 - accuracy: 0.5694 - scaled_graph_loss: 0.0257 - val_loss: 1.3099 - val_accuracy: 0.5905\n",
      "Epoch 116/200\n",
      "5/5 [==============================] - 1s 227ms/step - loss: 1.1480 - accuracy: 0.5713 - scaled_graph_loss: 0.0249 - val_loss: 1.3066 - val_accuracy: 0.5919\n",
      "Epoch 117/200\n",
      "5/5 [==============================] - 1s 194ms/step - loss: 1.1302 - accuracy: 0.5679 - scaled_graph_loss: 0.0253 - val_loss: 1.3026 - val_accuracy: 0.5937\n",
      "Epoch 118/200\n",
      "5/5 [==============================] - 1s 137ms/step - loss: 1.1127 - accuracy: 0.5759 - scaled_graph_loss: 0.0240 - val_loss: 1.3002 - val_accuracy: 0.5942\n",
      "Epoch 119/200\n",
      "5/5 [==============================] - 1s 209ms/step - loss: 1.1154 - accuracy: 0.5697 - scaled_graph_loss: 0.0271 - val_loss: 1.2991 - val_accuracy: 0.5942\n",
      "Epoch 120/200\n",
      "5/5 [==============================] - 1s 187ms/step - loss: 1.0834 - accuracy: 0.5843 - scaled_graph_loss: 0.0245 - val_loss: 1.2963 - val_accuracy: 0.5951\n",
      "Epoch 121/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 1.1061 - accuracy: 0.5903 - scaled_graph_loss: 0.0258 - val_loss: 1.2935 - val_accuracy: 0.5965\n",
      "Epoch 122/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 1.0833 - accuracy: 0.5821 - scaled_graph_loss: 0.0254 - val_loss: 1.2900 - val_accuracy: 0.5970\n",
      "Epoch 123/200\n",
      "5/5 [==============================] - 1s 175ms/step - loss: 1.1348 - accuracy: 0.5637 - scaled_graph_loss: 0.0248 - val_loss: 1.2858 - val_accuracy: 0.5988\n",
      "Epoch 124/200\n",
      "5/5 [==============================] - 1s 170ms/step - loss: 1.0713 - accuracy: 0.5912 - scaled_graph_loss: 0.0252 - val_loss: 1.2819 - val_accuracy: 0.5997\n",
      "Epoch 125/200\n",
      "5/5 [==============================] - 1s 176ms/step - loss: 1.0583 - accuracy: 0.5960 - scaled_graph_loss: 0.0277 - val_loss: 1.2799 - val_accuracy: 0.6006\n",
      "Epoch 126/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 1.0950 - accuracy: 0.6009 - scaled_graph_loss: 0.0253 - val_loss: 1.2770 - val_accuracy: 0.5993\n",
      "Epoch 127/200\n",
      "5/5 [==============================] - 1s 163ms/step - loss: 1.1018 - accuracy: 0.5771 - scaled_graph_loss: 0.0259 - val_loss: 1.2736 - val_accuracy: 0.6020\n",
      "Epoch 128/200\n",
      "5/5 [==============================] - 1s 147ms/step - loss: 1.1109 - accuracy: 0.5796 - scaled_graph_loss: 0.0273 - val_loss: 1.2704 - val_accuracy: 0.6016\n",
      "Epoch 129/200\n",
      "5/5 [==============================] - 1s 142ms/step - loss: 1.0983 - accuracy: 0.5808 - scaled_graph_loss: 0.0266 - val_loss: 1.2668 - val_accuracy: 0.6034\n",
      "Epoch 130/200\n",
      "5/5 [==============================] - 1s 135ms/step - loss: 1.1054 - accuracy: 0.5490 - scaled_graph_loss: 0.0296 - val_loss: 1.2626 - val_accuracy: 0.6066\n",
      "Epoch 131/200\n",
      "5/5 [==============================] - 1s 136ms/step - loss: 1.0896 - accuracy: 0.6092 - scaled_graph_loss: 0.0295 - val_loss: 1.2595 - val_accuracy: 0.6080\n",
      "Epoch 132/200\n",
      "5/5 [==============================] - 1s 148ms/step - loss: 1.0911 - accuracy: 0.5874 - scaled_graph_loss: 0.0292 - val_loss: 1.2571 - val_accuracy: 0.6076\n",
      "Epoch 133/200\n",
      "5/5 [==============================] - 1s 149ms/step - loss: 1.1144 - accuracy: 0.5697 - scaled_graph_loss: 0.0279 - val_loss: 1.2532 - val_accuracy: 0.6094\n",
      "Epoch 134/200\n",
      "5/5 [==============================] - 1s 140ms/step - loss: 1.0619 - accuracy: 0.5921 - scaled_graph_loss: 0.0314 - val_loss: 1.2494 - val_accuracy: 0.6103\n",
      "Epoch 135/200\n",
      "5/5 [==============================] - 1s 152ms/step - loss: 1.0882 - accuracy: 0.5957 - scaled_graph_loss: 0.0283 - val_loss: 1.2506 - val_accuracy: 0.6094\n",
      "Epoch 136/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 1.0127 - accuracy: 0.6250 - scaled_graph_loss: 0.0296 - val_loss: 1.2510 - val_accuracy: 0.6090\n",
      "Epoch 137/200\n",
      "5/5 [==============================] - 1s 157ms/step - loss: 1.0254 - accuracy: 0.6049 - scaled_graph_loss: 0.0278 - val_loss: 1.2501 - val_accuracy: 0.6103\n",
      "Epoch 138/200\n",
      "5/5 [==============================] - 1s 145ms/step - loss: 1.0017 - accuracy: 0.6117 - scaled_graph_loss: 0.0298 - val_loss: 1.2472 - val_accuracy: 0.6108\n",
      "Epoch 139/200\n",
      "5/5 [==============================] - 1s 155ms/step - loss: 1.0102 - accuracy: 0.6226 - scaled_graph_loss: 0.0277 - val_loss: 1.2472 - val_accuracy: 0.6117\n",
      "Epoch 140/200\n",
      "5/5 [==============================] - 1s 187ms/step - loss: 1.0174 - accuracy: 0.6061 - scaled_graph_loss: 0.0314 - val_loss: 1.2470 - val_accuracy: 0.6127\n",
      "Epoch 141/200\n",
      "5/5 [==============================] - 1s 175ms/step - loss: 1.0487 - accuracy: 0.6027 - scaled_graph_loss: 0.0279 - val_loss: 1.2464 - val_accuracy: 0.6131\n",
      "Epoch 142/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 1.0059 - accuracy: 0.5976 - scaled_graph_loss: 0.0290 - val_loss: 1.2446 - val_accuracy: 0.6131\n",
      "Epoch 143/200\n",
      "5/5 [==============================] - 1s 209ms/step - loss: 0.9457 - accuracy: 0.6522 - scaled_graph_loss: 0.0272 - val_loss: 1.2440 - val_accuracy: 0.6131\n",
      "Epoch 144/200\n",
      "5/5 [==============================] - 1s 176ms/step - loss: 1.0196 - accuracy: 0.6143 - scaled_graph_loss: 0.0281 - val_loss: 1.2449 - val_accuracy: 0.6136\n",
      "Epoch 145/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/5 [==============================] - 1s 222ms/step - loss: 1.0264 - accuracy: 0.6045 - scaled_graph_loss: 0.0281 - val_loss: 1.2458 - val_accuracy: 0.6136\n",
      "Epoch 146/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 0.9464 - accuracy: 0.6266 - scaled_graph_loss: 0.0315 - val_loss: 1.2463 - val_accuracy: 0.6136\n",
      "Epoch 147/200\n",
      "5/5 [==============================] - 1s 191ms/step - loss: 1.0403 - accuracy: 0.5913 - scaled_graph_loss: 0.0275 - val_loss: 1.2475 - val_accuracy: 0.6127\n",
      "Epoch 148/200\n",
      "5/5 [==============================] - 1s 231ms/step - loss: 1.0299 - accuracy: 0.6055 - scaled_graph_loss: 0.0302 - val_loss: 1.2493 - val_accuracy: 0.6127\n",
      "Epoch 149/200\n",
      "5/5 [==============================] - 1s 174ms/step - loss: 1.0777 - accuracy: 0.5722 - scaled_graph_loss: 0.0313 - val_loss: 1.2508 - val_accuracy: 0.6127\n",
      "Epoch 150/200\n",
      "5/5 [==============================] - 1s 265ms/step - loss: 1.0012 - accuracy: 0.6296 - scaled_graph_loss: 0.0288 - val_loss: 1.2516 - val_accuracy: 0.6131\n",
      "Epoch 151/200\n",
      "5/5 [==============================] - 1s 256ms/step - loss: 0.9506 - accuracy: 0.6113 - scaled_graph_loss: 0.0292 - val_loss: 1.2499 - val_accuracy: 0.6136\n",
      "Epoch 152/200\n",
      "5/5 [==============================] - 1s 250ms/step - loss: 1.0039 - accuracy: 0.6003 - scaled_graph_loss: 0.0286 - val_loss: 1.2448 - val_accuracy: 0.6145\n",
      "Epoch 153/200\n",
      "5/5 [==============================] - 1s 239ms/step - loss: 0.9514 - accuracy: 0.6343 - scaled_graph_loss: 0.0299 - val_loss: 1.2397 - val_accuracy: 0.6154\n",
      "Epoch 154/200\n",
      "5/5 [==============================] - 1s 199ms/step - loss: 0.9848 - accuracy: 0.6177 - scaled_graph_loss: 0.0309 - val_loss: 1.2356 - val_accuracy: 0.6177\n",
      "Epoch 155/200\n",
      "5/5 [==============================] - 1s 163ms/step - loss: 0.9157 - accuracy: 0.6665 - scaled_graph_loss: 0.0314 - val_loss: 1.2342 - val_accuracy: 0.6177\n",
      "Epoch 156/200\n",
      "5/5 [==============================] - 1s 160ms/step - loss: 0.9497 - accuracy: 0.6200 - scaled_graph_loss: 0.0301 - val_loss: 1.2320 - val_accuracy: 0.6177\n",
      "Epoch 157/200\n",
      "5/5 [==============================] - 1s 172ms/step - loss: 0.9808 - accuracy: 0.6151 - scaled_graph_loss: 0.0309 - val_loss: 1.2295 - val_accuracy: 0.6200\n",
      "Epoch 158/200\n",
      "5/5 [==============================] - 1s 181ms/step - loss: 0.9169 - accuracy: 0.6518 - scaled_graph_loss: 0.0283 - val_loss: 1.2264 - val_accuracy: 0.6219\n",
      "Epoch 159/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 1.0104 - accuracy: 0.6188 - scaled_graph_loss: 0.0289 - val_loss: 1.2250 - val_accuracy: 0.6228\n",
      "Epoch 160/200\n",
      "5/5 [==============================] - 1s 180ms/step - loss: 0.9568 - accuracy: 0.5875 - scaled_graph_loss: 0.0311 - val_loss: 1.2251 - val_accuracy: 0.6219\n",
      "Epoch 161/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 0.9131 - accuracy: 0.6352 - scaled_graph_loss: 0.0303 - val_loss: 1.2244 - val_accuracy: 0.6219\n",
      "Epoch 162/200\n",
      "5/5 [==============================] - 1s 168ms/step - loss: 0.9322 - accuracy: 0.6390 - scaled_graph_loss: 0.0308 - val_loss: 1.2250 - val_accuracy: 0.6223\n",
      "Epoch 163/200\n",
      "5/5 [==============================] - 1s 191ms/step - loss: 0.9138 - accuracy: 0.6420 - scaled_graph_loss: 0.0309 - val_loss: 1.2265 - val_accuracy: 0.6223\n",
      "Epoch 164/200\n",
      "5/5 [==============================] - 1s 161ms/step - loss: 0.9189 - accuracy: 0.6483 - scaled_graph_loss: 0.0310 - val_loss: 1.2288 - val_accuracy: 0.6214\n",
      "Epoch 165/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 0.9210 - accuracy: 0.6330 - scaled_graph_loss: 0.0331 - val_loss: 1.2299 - val_accuracy: 0.6228\n",
      "Epoch 166/200\n",
      "5/5 [==============================] - 1s 201ms/step - loss: 0.9685 - accuracy: 0.6292 - scaled_graph_loss: 0.0329 - val_loss: 1.2324 - val_accuracy: 0.6251\n",
      "Epoch 167/200\n",
      "5/5 [==============================] - 1s 260ms/step - loss: 0.9593 - accuracy: 0.6231 - scaled_graph_loss: 0.0320 - val_loss: 1.2372 - val_accuracy: 0.6251\n",
      "Epoch 168/200\n",
      "5/5 [==============================] - 1s 186ms/step - loss: 0.9453 - accuracy: 0.6082 - scaled_graph_loss: 0.0301 - val_loss: 1.2409 - val_accuracy: 0.6260\n",
      "Epoch 169/200\n",
      "5/5 [==============================] - 1s 174ms/step - loss: 1.0013 - accuracy: 0.6015 - scaled_graph_loss: 0.0313 - val_loss: 1.2456 - val_accuracy: 0.6247\n",
      "Epoch 170/200\n",
      "5/5 [==============================] - 1s 225ms/step - loss: 0.9140 - accuracy: 0.6605 - scaled_graph_loss: 0.0311 - val_loss: 1.2488 - val_accuracy: 0.6228\n",
      "Epoch 171/200\n",
      "5/5 [==============================] - 1s 184ms/step - loss: 0.8999 - accuracy: 0.6485 - scaled_graph_loss: 0.0295 - val_loss: 1.2475 - val_accuracy: 0.6237\n",
      "Epoch 172/200\n",
      "5/5 [==============================] - 1s 163ms/step - loss: 0.9913 - accuracy: 0.6180 - scaled_graph_loss: 0.0299 - val_loss: 1.2500 - val_accuracy: 0.6242\n",
      "Epoch 173/200\n",
      "5/5 [==============================] - 1s 229ms/step - loss: 0.9542 - accuracy: 0.6138 - scaled_graph_loss: 0.0290 - val_loss: 1.2513 - val_accuracy: 0.6237\n",
      "Epoch 174/200\n",
      "5/5 [==============================] - 1s 283ms/step - loss: 0.9251 - accuracy: 0.6392 - scaled_graph_loss: 0.0309 - val_loss: 1.2524 - val_accuracy: 0.6247\n",
      "Epoch 175/200\n",
      "5/5 [==============================] - 1s 277ms/step - loss: 0.9016 - accuracy: 0.6572 - scaled_graph_loss: 0.0321 - val_loss: 1.2525 - val_accuracy: 0.6260\n",
      "Epoch 176/200\n",
      "5/5 [==============================] - 1s 271ms/step - loss: 0.9267 - accuracy: 0.6182 - scaled_graph_loss: 0.0311 - val_loss: 1.2514 - val_accuracy: 0.6260\n",
      "Epoch 177/200\n",
      "5/5 [==============================] - 1s 231ms/step - loss: 0.8702 - accuracy: 0.6715 - scaled_graph_loss: 0.0307 - val_loss: 1.2500 - val_accuracy: 0.6265\n",
      "Epoch 178/200\n",
      "5/5 [==============================] - 1s 179ms/step - loss: 0.8859 - accuracy: 0.6498 - scaled_graph_loss: 0.0300 - val_loss: 1.2444 - val_accuracy: 0.6260\n",
      "Epoch 179/200\n",
      "5/5 [==============================] - 1s 232ms/step - loss: 0.9165 - accuracy: 0.6484 - scaled_graph_loss: 0.0306 - val_loss: 1.2410 - val_accuracy: 0.6265\n",
      "Epoch 180/200\n",
      "5/5 [==============================] - 1s 269ms/step - loss: 0.8989 - accuracy: 0.6480 - scaled_graph_loss: 0.0308 - val_loss: 1.2395 - val_accuracy: 0.6260\n",
      "Epoch 181/200\n",
      "5/5 [==============================] - 1s 280ms/step - loss: 0.9084 - accuracy: 0.6570 - scaled_graph_loss: 0.0303 - val_loss: 1.2380 - val_accuracy: 0.6270\n",
      "Epoch 182/200\n",
      "5/5 [==============================] - 1s 184ms/step - loss: 0.8927 - accuracy: 0.6529 - scaled_graph_loss: 0.0321 - val_loss: 1.2398 - val_accuracy: 0.6293\n",
      "Epoch 183/200\n",
      "5/5 [==============================] - 1s 269ms/step - loss: 0.9331 - accuracy: 0.6413 - scaled_graph_loss: 0.0324 - val_loss: 1.2425 - val_accuracy: 0.6274\n",
      "Epoch 184/200\n",
      "5/5 [==============================] - 1s 232ms/step - loss: 0.8546 - accuracy: 0.6809 - scaled_graph_loss: 0.0297 - val_loss: 1.2477 - val_accuracy: 0.6283\n",
      "Epoch 185/200\n",
      "5/5 [==============================] - 1s 195ms/step - loss: 0.8826 - accuracy: 0.6345 - scaled_graph_loss: 0.0327 - val_loss: 1.2524 - val_accuracy: 0.6274\n",
      "Epoch 186/200\n",
      "5/5 [==============================] - 1s 234ms/step - loss: 0.8374 - accuracy: 0.6524 - scaled_graph_loss: 0.0321 - val_loss: 1.2602 - val_accuracy: 0.6288\n",
      "Epoch 187/200\n",
      "5/5 [==============================] - 1s 240ms/step - loss: 0.9199 - accuracy: 0.6291 - scaled_graph_loss: 0.0331 - val_loss: 1.2633 - val_accuracy: 0.6279\n",
      "Epoch 188/200\n",
      "5/5 [==============================] - 1s 198ms/step - loss: 0.8474 - accuracy: 0.6932 - scaled_graph_loss: 0.0315 - val_loss: 1.2667 - val_accuracy: 0.6270\n",
      "Epoch 189/200\n",
      "5/5 [==============================] - 1s 162ms/step - loss: 0.8857 - accuracy: 0.6527 - scaled_graph_loss: 0.0326 - val_loss: 1.2682 - val_accuracy: 0.6251\n",
      "Epoch 190/200\n",
      "5/5 [==============================] - 1s 167ms/step - loss: 0.8189 - accuracy: 0.6870 - scaled_graph_loss: 0.0335 - val_loss: 1.2717 - val_accuracy: 0.6251\n",
      "Epoch 191/200\n",
      "5/5 [==============================] - 1s 157ms/step - loss: 0.9053 - accuracy: 0.6332 - scaled_graph_loss: 0.0321 - val_loss: 1.2757 - val_accuracy: 0.6233\n",
      "Epoch 192/200\n",
      "5/5 [==============================] - 1s 156ms/step - loss: 0.9003 - accuracy: 0.6519 - scaled_graph_loss: 0.0333 - val_loss: 1.2747 - val_accuracy: 0.6256\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 193/200\n",
      "5/5 [==============================] - 1s 165ms/step - loss: 0.8634 - accuracy: 0.6420 - scaled_graph_loss: 0.0301 - val_loss: 1.2704 - val_accuracy: 0.6270\n",
      "Epoch 194/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 0.8267 - accuracy: 0.6727 - scaled_graph_loss: 0.0314 - val_loss: 1.2641 - val_accuracy: 0.6274\n",
      "Epoch 195/200\n",
      "5/5 [==============================] - 1s 172ms/step - loss: 0.8430 - accuracy: 0.6941 - scaled_graph_loss: 0.0338 - val_loss: 1.2606 - val_accuracy: 0.6283\n",
      "Epoch 196/200\n",
      "5/5 [==============================] - 1s 177ms/step - loss: 0.8967 - accuracy: 0.6375 - scaled_graph_loss: 0.0320 - val_loss: 1.2575 - val_accuracy: 0.6316\n",
      "Epoch 197/200\n",
      "5/5 [==============================] - 1s 164ms/step - loss: 0.8748 - accuracy: 0.6446 - scaled_graph_loss: 0.0315 - val_loss: 1.2558 - val_accuracy: 0.6320\n",
      "Epoch 198/200\n",
      "5/5 [==============================] - 1s 190ms/step - loss: 0.9019 - accuracy: 0.6390 - scaled_graph_loss: 0.0323 - val_loss: 1.2531 - val_accuracy: 0.6316\n",
      "Epoch 199/200\n",
      "5/5 [==============================] - 1s 139ms/step - loss: 0.7997 - accuracy: 0.6777 - scaled_graph_loss: 0.0342 - val_loss: 1.2514 - val_accuracy: 0.6325\n",
      "Epoch 200/200\n",
      "5/5 [==============================] - 1s 247ms/step - loss: 0.9136 - accuracy: 0.6405 - scaled_graph_loss: 0.0328 - val_loss: 1.2526 - val_accuracy: 0.6320\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14fe35610>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graph_reg_model.fit(myTrain.batch(128), epochs=200, verbose=1, validation_data=myTest.batch(128),\n",
    "          callbacks=[TensorBoard(log_dir='/tmp/regularization')])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml-book-4",
   "language": "python",
   "name": "ml-book-4"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
